{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Helicone\n",
    "\n",
    "This notebook demonstrates how to integrate Helicone with Ragas for monitoring and evaluating RAG (Retrieval-Augmented Generation) systems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "Before you begin, make sure you have a Helicone account and API key:\n",
    "\n",
    "1. Log into [Helicone](https://www.helicone.ai) or create an account if you don't have one.\n",
    "2. Once logged in, navigate to the [Developer section](https://helicone.ai/developer) to generate an API key.\n",
    "\n",
    "**Note**: Make sure to generate a write-only API key. For more information on Helicone authentication, refer to the [Helicone Auth documentation](https://docs.helicone.ai/getting-started/helicone-api-keys).\n",
    "\n",
    "Store your Helicone API key securely, as you'll need it for the integration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "First, let's install the required packages and set up our environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install datasets ragas openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from datasets import Dataset\n",
    "from ragas import evaluate\n",
    "from ragas.metrics import faithfulness, answer_relevancy, context_precision\n",
    "from ragas.integrations.helicone import helicone_config  # import helicone_config\n",
    "\n",
    "\n",
    "# Set up Helicone\n",
    "helicone_config.api_key = (\n",
    "    \"your_helicone_api_key_here\"  # Replace with your actual Helicone API key\n",
    ")\n",
    "os.environ[\n",
    "    \"OPENAI_API_KEY\"\n",
    "] = \"your_openai_api_key_here\"  # Replace with your actual OpenAI API key\n",
    "\n",
    "# Verify Helicone API key is set\n",
    "if HELICONE_API_KEY == \"your_helicone_api_key_here\":\n",
    "    raise ValueError(\n",
    "        \"Please replace 'your_helicone_api_key_here' with your actual Helicone API key.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare Data\n",
    "\n",
    "Let's prepare some sample data for our RAG system evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_samples = {\n",
    "    \"question\": [\"When was the first Super Bowl?\", \"Who has won the most Super Bowls?\"],\n",
    "    \"answer\": [\n",
    "        \"The first Super Bowl was held on January 15, 1967.\",\n",
    "        \"The New England Patriots have won the most Super Bowls, with six championships.\",\n",
    "    ],\n",
    "    \"contexts\": [\n",
    "        [\n",
    "            \"The First AFL–NFL World Championship Game, later known as Super Bowl I, was played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles, California.\"\n",
    "        ],\n",
    "        [\n",
    "            \"As of 2021, the New England Patriots have won the most Super Bowls with six championships, all under the leadership of quarterback Tom Brady and head coach Bill Belichick.\"\n",
    "        ],\n",
    "    ],\n",
    "    \"ground_truth\": [\n",
    "        \"The first Super Bowl was held on January 15, 1967.\",\n",
    "        \"The New England Patriots have won the most Super Bowls, with six championships as of 2021.\",\n",
    "    ],\n",
    "}\n",
    "\n",
    "dataset = Dataset.from_dict(data_samples)\n",
    "print(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluate with Ragas\n",
    "\n",
    "Now, let's use Ragas to evaluate our RAG system. Helicone will automatically log the API calls made during this evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate using Ragas\n",
    "score = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])\n",
    "\n",
    "# Display results\n",
    "print(score.to_pandas())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Viewing Results in Helicone\n",
    "\n",
    "The API calls made during the Ragas evaluation are automatically logged in Helicone. You can view these logs in the Helicone dashboard to get insights into the performance and behavior of your RAG system.\n",
    "\n",
    "To view the results:\n",
    "1. Go to the [Helicone dashboard](https://www.helicone.ai/dashboard)\n",
    "2. Navigate to the 'Requests' section\n",
    "3. You should see the API calls made during the Ragas evaluation\n",
    "\n",
    "You can analyze these logs to understand:\n",
    "- The number of API calls made during evaluation\n",
    "- The performance of each call (latency, tokens used, etc.)\n",
    "- Any errors or issues that occurred during the evaluation\n",
    "\n",
    "This integration allows you to combine the power of Ragas for RAG system evaluation with Helicone's robust monitoring and analytics capabilities."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
